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Abstract 

Based on the Lie theoretical methods of algebraic Fourier transfor- 
mation, we classify singular vectors of diagonally embedded generalized 
Verma modules for orthogonal Lie algebra and its conformal parabolic 
subalgebra with commutative nilradical, thereby realizing the diagonal 
branching rules. In particular, we construct the singular vectors for all 
one dimensional inducing representations, exploiting recursive structure 
for the generalized hypergeometric function 3.F2. As a geometrical con- 
sequence, we classify bilinear conformally invariant differential operators 
acting on homogeneous line bundles and realize their dependence on in- 
ducing representation theoretical parameters in an explicit way. 
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1 Introduction and Motivation 

The subject of our article has its motivation in the Lie theory for finite 
dimensional simple Lie algebras applied to the problem of branching rules 
and composition structure of generalized Verma modules, and dually in 
the geometrical problem related to the construction of invariant bilin- 
ear differential operators or Rankin-Cohen-like brackets for automorphic 
forms associated to orthogonal groups. 

The bilinear invariant differential operators, as the simplest repre- 
sentatives of multilinear invariant differential operators organized in an 
Aoo-homotopical structure, appear in a wide range of Lie theoretic ap- 
plications. For example, the classical Rankin-Cohen brackets realized by 
holomorphic SL(2, R)-invariant bilinear differential operators on the up- 
per half plane H are devised, originally in a number theoretic context, 
to produce from a given pair of modular forms another modular form. 
They turn out to be intertwining operators producing ring structure on 
SL(2, R) holomorphic discrete series representations, and can be analyt- 
ically continued to the full range of inducing characters. Consequently, 
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such operators were constructed in several specific situations of interest 
related to Jacobi forms, Siegel modular forms, holomorphic discrete series 
of causal symmetric spaces of Cayley type, etc., [5], [8], [15] . 

The main reason behind the underlying classification scheme for such 
class of operators is inspired by geometrical analysis on manifolds with, 
e.g., the conformal structure, and related PDE problems of geometrical 
origin. For M a smooth (or, complex) manifold equipped with a filtration 
of its tangent bundle TM, V a smooth (or holomorphic) vector bundle 
on M and J k V the weighted jet bundle, a bilinear differential pairing 
between sections of the bundle V and sections of the bundle VV to sections 
of a bundle 3^ is a sheaf homomorphism 

B : J k V®J l W -»■ y. 

In the case M — G/P is a generalized flag manifold, a pairing is called in- 
variant if it commutes with the action of G on sections of the homogeneous 
vector bundles V, VV, V- Denoting V, W, Y the inducing P-representations 
of homogeneous vector bundles V, VV, y, G-invariant differential pairings 
can be algebraically characterized as the space 

Hom u( - s) (M(s, P, Y), (U(g) g W(5)) ®uww(to (V v ® W v )). (1) 

In other words, the former geometrical problem for finding bilinear invari- 
ant differential operators on G/P acting on induced representations turns 
into a Lie algebraic problem of the characterization of homomorphisms of 
generalized Verma modules. In the geometrical context of flag manifolds 
and general curved manifolds with parabolic structure, a classification of 
first order bilinear differential operators for parabolic subalgebras with 
commutative nilradicals (so called AHS structures) was completed in 1 1 5] . 
One of the main applications of bilinear differential operators is that they 
act via invariant cup product as symmetries of invariant differential oper- 
ators, see e.g. [J] for the case of conformally invariant Laplace operator. 

Yet another approach to these questions is purely analytical and con- 
sists of meromorphic continuation of invariant distributions given by a 
multilinear form on the principal series representations. For example, a 
class of G = SOo(n+l, 1,R) (i.e., conformally )-invariant linear and bilin- 
ear differential operators was constructed as residues of meromorphically 
continued invariant trilinear form on principal series representations in- 
duced from characters, see [2]. 

To summarize, our article contains a general Lie theoretic classifica- 
tion of Rankin- Cohen-like brackets for the couple of real orthogonal Lie 
algebra so(n + 1,1, R) and its conformal parabolic Lie subalgebra, and 
their explicit - in the sense of dependence on representation theoretical 
parameters - construction for characters as inducing representations. 

The structure of our article goes as follows. As already mentioned, we 
first reformulate the existence of invariant bilinear differential operators 
(or equivalently, Rankin-Cohen-like brackets) in terms of purely abstract 
Lie theoretic classification scheme for diagonal branching rules of gen- 
eralized Verma modules, associated to the real orthogonal Lie algebra 
so(n + 1, 1,R) and its conformal parabolic Lie subalgebra p. The reason 
behind the choice for this parabolic subalgebra is its fundamental prop- 
erty of having the commutative nilradical. Consequently, the branching 
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problem takes value in the Grothendieck group K(O p ) of the Bernstein- 
Gelfand-Gelfand parabolic category O v . Here the main device are char- 
acter formulas and their reduction in the branching problems, see Section 
2. The quantitative part of the problem consists of the construction of 
singular vectors, and is the content of Section 3. It is based on the proce- 
dure of rewriting the representation theoretical action in the Fourier dual 
picture, where the positive nilradical of p is acting on symmetric algebra 
of the (commutative) opposite nilradical. This action produces the four 
term functional equation for singular vectors, and its solution is techni- 
cally the most difficult part with both analytic and combinatorial aspects 
arising from generalized hypergeometric equation. The last Section 4 is 
a direct consequence of Sections 2, 3 and determines the explicit formulas 
for bilinear conformally invariant differential operators representing these 
singular vectors. 

Throughout the article we denote by N the set of natural numbers 
including zero. 

2 Branching problem and algebraic Fourier 
transformation (F-method) 

In the present section we briefly review the tool developed in [TO], 
which allows to explicitly realize the branching problem for the semisimple 
symmetric pair (jjffljj, diag(g)) and the generalized Verma W(0ffl£))-modules 
induced from characters of the parabolic subalgebra p ffi p C (g ffi fl). 
Throughout the article we use the notation diag for a diagonally embedded 
subalgebra, e.g., diag : g t -* (jj ffi g). 

This approach is termed F-method and is based on the analytical tool 
of algebraic Fourier transformation on the commutative nilradical n of p. 
It allows to find singular vectors in generalized Verma modules, exploiting 
algebraic Fourier transform and classical invariant theory. Its main ad- 
vantage is the conversion of combinatorially complicated problem in the 
universal enveloping algebra of a Lie algebra into the system of partial or 
ordinary special differential equations acting on a polynomial ring. 

Let G be a connected real reductive Lie group with the Lie algebra 
0, P C G a parabolic subgroup and p its Lie algebra, p = 1 © n the 
Levi decomposition of p and ft- its opposite nilradical, g — n_ ©p. The 
corresponding Lie groups are denoted N-,L,N. Denoting the fibration 
p : G ->■ G/P, M := p(N- ■ P) is the big Schubert cell of G/P and the 
exponential map 

n_ ->■ M, X ^ exp(X) ■ o G G/P, o := e ■ P G G/P, e G G. 

gives the canonical identification of the vector space n_ with M . 

Given a complex finite dimensional P-module V, we consider the in- 
duced representation ty of G on the space Jndp(V) of smooth sections of 
the homogeneous vector bundle G x p V — > G/P, 

Jnd|(V) = C°°(G, V) P := {/ G C 00 (G,V)\f(g-p)=p- 1 -f(g), 9 eG, P e P}. 
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Let U(gc) denote the universal enveloping algebra of the complexified 
Lie algebra Qc- Let V v be the dual (contragredient) representation to V. 
The generalized Verma module M^(V V ) is defined by 

M~l(V v ) :=W(g) ® M( p)V v , 

and there is a (0, P)-invariant natural pairing between Ind^(V) and M^(Y v ), 
described as follows. Let V'(G/P) <g> V v be the space of all distributions 
on G/P with values in V v . The evaluation defines a canonical equivari- 
ant pairing between Ind^(V) and T>'(G/P) ® V v , and this restricts to the 
pairing 

Ind^ (V) x V' [o] (G/P) <g> V v -> C, (2) 

where T>'(G/P)[ ] ® V v denotes the space of distributions supported at 
the base point o £ G/P. As shown in [3], the space X>[ ](G/P) ® V v 
can be identified, as an W(jj)-module, with the generalized Verma module 
-M|(V V ). 

Moreover, the space of G-equivariant differential operators from Ind? (V) 
to Indp(V') is isomorphic to the space of (g, P)-homomorphisms between 
A1?(V' V ) and A4?(V V ) for two inducing representations V and V' of P. 
The homomorphisms of generalized Verma modules are determined by 
their singular vectors, and the F-method translates searching for singular 
vectors to the study of distributions on G/P supported at the origin, and 
consequently to the problem of solution space for the system of partial 
differential equations acting on polynomials Pol(n) on n. 

The representation ir of G on Indp(V) has the infinitesimal represen- 
tation d-k of 0c, in the non-compact picture acting on functions on the big 
Schubert cell ft_ ~ M C G/P with values in V. The latter representation 
space can be identified via exponential map with G°°(n_, V). The action 
(1tt(Z) of elements Z € ft on G°°(ft_, V) is realized by vector fields on n_ 
with coefficients in Pol(ft_ ) ® End V, see [13] . 

By the Poincare-Birkhoff-Witt theorem, the generalized Verma mod- 
ule Ai~ l (V v ) is isomorphic as an [-module to the algebraic Weyl algebra of 
^--invariant of holomorphic differential operators on n_, W(n_) ® V v ~ 
Diff i y_(n-)<g>V v . In the special case when ft- is commutative, Diff ^ (n_ ) 
is the space of holomorphic differential operators on n_ with constant co- 
efficients. Moreover, the operators dn v (X), X £ jj, are realized as differ- 
ential operators on ft_ with coefficients in End(V v ). The application of 
Fourier transform on fi_ gives the identification of the generalized Verma 
module Diff^ (ft-) <g> V v with the space Pol(n) ® V v , and the action 
dit v of on Diffjy (ft-) ® V v translates to the action (dn V ) F of on 
Pol(ft) CS> V v and is realized again by differential operators with values in 
End(V v ). The explicit form of {d-H V ) F (X) is easy to compute by Fourier 
transform from the explicit form of dir v . 

A generalization of the previous framework is based on P C G intro- 
duced above, and yet another couple of Lie groups P' C G compatible 
with (G, P) in the sense that G' C G is a reductive subgroup of G and 
P' = P n G' is a parabolic subgroup of G'. The Lie algebras of G',P' 
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are denoted by g',p'. In this case, n' := ft n g' is the nilradical of p', and 
L' = L n G' is the Levi subgroup of P' . We are interested in the branch- 
ing problem for generalized Verma modules .M~(V v ) under the restriction 
from jj to g'. 

Definition 2.1 Let V be an irreducible P-module. Let us define the L' - 
module 

M S(y v f := {v G X?(V v )|d7r v (Z)v = for all Z G ft'}. (3) 

Note that for G / G' , the set .M~(V v ) n is infinite dimensional com- 
pletely reducible I'-module, while for G = G' the set Af?(V v ) fi ' is finite 
dimensional. An irreducible Z'-submodule W v of M~ l (Y v ) n gives an in- 
jective W(g^)-homomorphism from Af?,(W v ) to A4jjj(V v ). Dually, we get 
an equivariant differential operator acting from Indp(V) to Indp, (W). 

Using the F-method, the space of Z'-singular vectors M~ l (Y v ) n is 
realized in the ring of polynomials on ft valued in V v and equipped with 
the action of (div V ) F . 

Definition 2.2 We define 

Sol(g,g,Y v ) := {/ G Pol(ft) <g> V v | {di y ) F {Z)f = for all Z G ft'}. (4) 

Then the inverse Fourier transform gives an L'-isomorphism 

SolC Q ,g';Y V )^Ml(Y v f. (5) 

An explicit form of the action (d% v ) F (Z) leads to a system of differential 
equation for elements in Sol. The transition from A/f?(V v ) n to Sol allows 
to transform the algebraic problem of computation of singular vectors in 
generalized Verma modules into analytic problem of solving the system of 
partial differential equations. 

In the dual language of differential operators acting on principal se- 
ries representation, the set of G"-intertwining differential operators from 
Ind?(V) to Ind?, (V') is in bijective correspondence with the space of all 
(fl',P')-homomorphisms from Ml',(Y' v ) to Af?(V v ). 

3 Abstract characterization of diagonal 
branching rules applied to generalized Verma 
modules for so(n + 1, 1, R) 

The present section contains qualitative results on the diagonal branching 
rules for so(n + 1, 1, R) applied to generalized Verma modules. 

Let n G N such that n > 3. Throughout the article g denotes the 
real Lie algebra so(n + 1,1, R) of the connected and simply connected 
simple Lie group G = S0 o {n + 1, 1,R). Let p be its maximal parabolic 
subalgebra p = [ k n, in the Dynkin diagrammatic notation for parabolic 
subalgebras given by omitting the first simple root of g. The Levi factor I 
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of p is isomorphic to so(n, I) xl and the commutative nilradical n (resp. 
the opposite nilradical n_) is isomorphic to R n . Let diag : (fj,p) ■— > 
(fl ffi 0i P ffi P) denote the diagonal embedding. 

The main task of the present article concerns the branching problem 
for the family of scalar generalized Verma U(g ffi g, p ffi p)-modules induced 
from characters of the center of 1 © [, with respect to diag(g,p). An 
inducing character xa,/j on pffip is determined by two complex characters 
Xm.Xa on [: 

Xa, m = (xa,Xm) : Pffip 

(f>i,P2) Xa(pi) ■ Xm(P2), (6) 

where the homomorphism (xa,Xm) quotients through the semisimple sub- 
algebra [p,p]ffi[p,p]. The generalized Verma (g ffi g, p ffi p)-module induced 
from character (xa,Xm) (^>A* G C) is 

M(g,p,Y x ,„) =Ma,,(9,))) = W(0 ffi 0) ® M(P ®p) (Ca®C m ), (7) 

where Ca ® C M is a 1-dimensional representation (xa,Xm) of p ffip. As a 
vector space, M\,ii(g,p) is isomorphic to the symmetric algebra S*(ti— ffi 
n_), where n_ ffi n_ is the vector complement of p ffi p in g ffi g. 

A way to resolve this branching problem abstractly is based on charac- 
ter identities for the restriction of M\ tll (g,p) to the diagonal subalgebra 
diag(g) with standard compatible parabolic subalgebra 

diag(p) := diag(g) n (p ffi p), p = I x n, diag(p) — diag{\) X diag(n). 

For a fixed choice of positive simple roots of g we denote by A + ( [ffi [) the set 
of weights dominant for [ffi [ and integral for [[, [] ffi [[, []. Let Va, m be a finite 
dimensional irreducible Iffil-module with highest weight (A,/x) £ A + ([ffi[), 
and likewise Vv be a finite dimensional representation of diag(V), A' G 
A + (diag(l)). Given a vector space V we denote 5*(V) = ffig S , ; (V) the 
symmetric tensor algebra on V. Let us extend the adjoint action of diag(\) 
on (n-ffin_)/((n-ffin_)ndia3(fl)) to S*((n_0lX-)/((n_©n_)ndiaff(fl))). 
Notice that we have an isomorphism 

(n_ ffi n_)/((n_ ffi n_) n diag(g)) ~ n_ 

of dia(?(()-quotient modules. We set 

m(A', (A,/i)) := 

Hom diagW (V\>, VA, M Uiag([) ffi S*((n_ ffin_)/((n_ ffin_) ndiac^g))). 

(8) 

Let us recall 

Theorem 3.1 ^Jiffi Theorem 3.9) Let (0,p), (g',p') be a compatible cou- 
ple of simple Lie algebras and their parabolic subalgebras, and suppose p is 
0' -compatible standard parabolic subalgebra of g, (A, /1) £ A + ([ffi [). Then 

1. m(A', < 00 for all A' G A + (0- 
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2. In the Grothendieck group of Bernstein-Gelfand-Gelfand parabolic 
category O p there is q' -isomorphism 

MxAlP)W- © m(A',(A,/x))A4 v (fl',p'). 

A'6A+(T') 

Applied to the case of our interest (jj,p) = (fl © 0,p © p), (b',P') = 
diag(g,p) = (diag(g),diag(p)), we see that the polynomial ring S*((n_ © 
n_)/((n_ ffin_) PI diag(g))) decomposes as dia<?([)-module on irreducibles 
with higher multiplicities. In particular, each dia<;(I)-module realized in 
homogeneity k polynomials also appears in polynomials of homogeneity 
(k + 2), k 6 N. As we already explained, we focus on the case of 1- 
dimensional inducing representations Va, m — Ca © C M as ([ © [)-modules 
and Vv — C„ as diap(I)-modules (\,fj,,v 6 C). The multiplicity for- 
mula then implies that a nontrivial homomorphism in (|8| occurs for each 
1-dimensional dzag([)-module in S*((n_ © n_)/((n_ © n_) PI diag(g))\ 
and it is a result in classical invariant theory (see [7], [14]) that for each 
even homogeneity there is just one 1-dimensional module. Because n_ 
is as (diag(V)/[diag(V),diag(V)])-modu\e isomorphic to the character C_i, 
the following relation holds true in the Grothendieck group of with 
p ~ diag(p): 

Corollary 3.2 For gffig = so(n+l, 1, R)ffiso(n+l, 1, R), diag(g) ~ so(n+ 
1, 1,R) with standard maximal parabolic subalgebras p ffi p,diag(p) given 
by omitting the first simple root in the corresponding Dynkin diagrams, 
m(v, (A, n)) = 1 if and only if v = \ + jj, — 2j,j £ N and m(y, (A, fi)) = 
otherwise. 

Consequently, in the Grothendieck group of the Bernstein-Gelfand- 
Gelfand parabolic category O p holds 

A+n-2j(9, p), 

where v — A + /i — 2j . 

Although we work in one specific signature (n + 1, 1), the results are 
easily extended to a real form of any signature. 

4 The construction of singular vectors for 
diagonal branching rules applied to gener- 
alized Verma modules for so(n + 1, 1, R) 

The rest of the article is devoted to the construction of singular vectors, 
whose abstract existence was concluded in the Section 3, Corollary |3.2l us- 
ing the tool reviewed in Section 2. This can be regarded as a quantitative 
part of our diagonal branching problem. 
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4.1 Description of the representation 

In this subsection we describe the representation of @ © g, acting upon the 
generalized Verma module 

Mx,„(8®9,P<$P) = "(effis) ® M(p(B p) (C AlM ), Ca, m = c A ® c„, 

in its Fourier image, i.e. apply the framework for the F-method explained 
in Section 2 to 

>U, M (0 0g,p©p) -MA(g,t))0M,(g,p). (9) 

The first goal is to describe the action of elements in the nilradical diag(n) 
of diag(p) in terms of differential operators acting on the Fourier image 
of A / !a,m(0 © 0iP © P)- This can be derived from the explicit form of the 
action on the induced representation realized in the non-compact picture, 
and it follows from (J5J) that the problem can be reduced to the question 
on each component of the tensor product separately. Let us consider the 
complex representation 7ta, A G C, of G = SO a (n + 1, 1,R), induced from 
the character p n> a x , p G P, acting on the one dimensional representation 
space Ca — C. Here a G A — R* is the abelian subgroup in the Langlands 
decomposition P = MAN, M = SO{n), N = R n . In other words, iv\ 
acts by left regular representation on Indp(CA). 

Let Xj be the coordinates with respect to the standard basis on n_, 
and j = 1, ... ,ti the coordinates on the Fourier transform of n_. We 
consider the family of differential operators 

QiW = + + x * d *)> 3 = 1, • • • ,n, (10) 

k 

PfW = i Q^A« + (A — E«)0 £i ) ,j = l,...,n, (11) 
where \x\ 2 = xf + ■ ■ ■ + x\, 

a« = dl + ■ ■ ■ + d 2 u 

is the Laplace operator of positive signature, dj = g|- and E 4 = ^ fc ^kd^ k 
is the Euler homogeneity operator (i G C the complex unit.) The following 
result is a routine computation: 

Lemma 4.1 ('101) Let us denote by Ej the standard basis elements ofn, 
j = 1, ... ,n. Then Ej G n are acting on C°°(n-, C_a) by 

dSA(S 3 )(s®u) = Qj(\)(s)®v, s G C°°(n-,C), v G C-a, (12) 

and the action of (dw)x ° n Pol[^i, . . . , £ n ] ® Ca is given by 

{d^liE^f ® v) = P/(A)(/) ® «, / G PolKi, ■ • • ,?»], »£C„ A v. (13) 

As for the action of remaining basis elements of in the Fourier image 
of the induced representation, the action of n_ is given by multiplication 
by coordinate functions, the standard basis elements of the simple part of 
the Levi factor [[, [] = so(n) act by differential operators 
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and the basis element of the Lie algebra of A acts as the homogeneity 
operator, E e = J27=i • 

In the Fourier image of the tensor product of two induced represen- 
tations in the non-compact realization on n_ © n_ with coordinates £j 
resp. Vi on the first resp. second copy of n_ in n_ © n_, the generators of 
the diagonal subalgebra diag(g) act on the representation Indp*p(CA. M ) 
induced from (xa,Xm) as 

1. Multiplication by 

(t i ®l) + (l®v j ), j = l,...,n (14) 

for the elements of diag(x\—), 

2. First order differential operators with linear coefficients 

Mff = (A4®1) + (1®M5) 

= (&% - (81 + 1® {v 3 d Ui - Uid Vj ), (15) 

i,j — 1, . . . ,n for the elements of the simple subalgebra of diag{\) 
and 

n 

E 5 <g>l + 1®E" = J2& 9 ^ ®l + l®"idvi), (16) 
i=i 

for the generator of diag([)/[dia(?(I),dia(7(I)], 

3. Second order differential operators with linear coefficients 

Pf(A,/i) = (P/(A)®l) + (l®P/(/x)) 
= i(i&A f + (A-E*)£y ®1 

(i^A' + Oi-E")^), (17) 

j = 1, . . . ,n for the elements diag(n). 

This completes the description of the first part of abstract procedure out- 
lined in Section 2 in the case of the diagonal branching problem of our 
interest. 

4.2 Reduction to a hypergeometric differential equa- 
tion in two variables 

It follows from the discussion in Sections 2, 3 that dia(;([)-modules induc- 
ing singular vectors for the diagonal branching rules are one dimensional. 
This means that they are annihilated by diag(l s ) = diag([l, []) ~ so(n, R), 
the simple part of the diagonal Levi factor diag(V) ~ so(n, R) x R. It fol- 
lows that the singular vectors are invariants of diag(V) acting diagonally 
on the algebra of polynomials on n_ © n_ as a [ © [-module. The follow- 
ing result is a special case of the first fundamental theorem in classical 
invariant theory, see e.g. [7], [14| . 
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Lemma 4.2 Let (V, <, >) be a finite dimensional real vector space with 
bilinear form <, > and SO(V) the Lie group of automorphisms of (V, < 
, >). Then the subalgebra of SO(V) -invariants in the complex polynomial 
algebra Pol[V © V] (SO(V) acting diagonally on V ®V) is polynomial 
algebra generated by < £,£ >, < £, v > and < v,v >. Here we use the 
convention that £ is a vector in the first component V of V © V and v in 
the second summand. 

In our case, the complex polynomial algebra is Pol[£i, ...,£„, 1/1, i/„] 
and we use the notation Pol[r, s, t] for the (complex) subalgebra of invari- 
ants: 

n 

r :=< £,v >=^iiVi, 

n 

s:=< £,£>=X)&&' 
i=l 

n 

t: = <u,u >=^UiUi. (18) 

i=l 

The task of the present subsection is to rewrite the operators i-^' "(A, /i) 
in the variables r,s,t, i.e. we reduce the action of P*' I/ (A,/i) from the 
polynomial ring to the ring of dia(/(n_)-invariants on ri- © n_. 
We compute 

d Vi r = fj, %r = Vi,d Vi s = 0, d St s = 2£i,d Vi t = 2u t , d St t = 0, (19) 
and 

% = Vid r + 2£ids, A s = tdr + 4rd r d s + 2nd s + 4sdf , i = 1, . . . , n. (20) 

Note that analogous formulas for d Vi , A" can be obtained from those for 
£ by applying the involution 

& < — ► Vi, s < — ► t, r < — ► r. (21) 

We also have for alH = 1, . . . , n 

E s d u = u 1 (E r + 2E s )d r + C,(2E r + IE* + 2)9,, (22) 

and so taking all together we arrive at the operators 

Pr s '\\,n) = fc(it# + (n + 2A - 2 - 2E s )d s - (E r + 2E* - n)d r ) 
+Vi{\sd 2 r + {n + 2^-2- 2E t )d t - (E r + 2E S - X)d r ) (23) 
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acting on complex polynomial algebra Pol[r, s, t], i = 1, . . . , n. A conse- 
quence of the system of equations (i — 1, . . . , n) is 

n 1 
P^'''*(X,H) := X) W*^) = + (n + 2A - 2 - 2E S )<9 S 

i=l 

-(E r + 2E* - /i)a r ) + r(i S a? + (n + 2 M - 2 - 2E*)9< - (E r + 2E S - \)d r ), 

n 1 
P;' S '*(A, M ) := X ^^ r ' S ''(A>M) = + + 2A - 2 - 2E S )9 S 

i=l 

-(E r + 2E* - /x)a r ) + t{^sd 2 r + (n + 2)i-2- 2E t )d t - (E r + 2E S - X)d r ). 

(24) 

Notice that the second equation follows from the first one by the action 
of involution 

A < — 7- /i, s < — > t, r < — > r. 

In what follows we construct a set of homogeneous polynomial solutions 
of Pj' s ' (A, fi), P/ s,t (A,p) solving the system {P[' s ' (A, fJ>)}i, i= 1, . . . ,n. 
The uniqueness of the solution for the generic values of the inducing pa- 
rameters implies the unique solution of the former system of PDEs. 

Notice that (|24[) is the system of differential operators preserving the 
space of homogeneous polynomials in the variables r, s, t, i.e. Pf' 8 ' (A, fj,), 
P;' S '*(A, fi) commute with E r ' s '* := E r + E s + E*. 

Let p = p(r, s, i) be a homogeneous polynomial of degree N, deg(p) = 
TV, and write 

JV , S t . jv -/ \ s t 

p = r p( — ,—)=r p(u, v), u := — , v := -, 
r r r r 

p(u,v)= (25) 

i,j|0<|i+j|<JV 

where p is the polynomial of degree TV. We have 

d s = -d u + —d r , dt = —dv + —dr 
r u r v 

and so the summands in (I24|l transform as 
±std 2 r = i/V(/V - l)uv, 

s(n + 2A - 2 - 2E S )9 S = (n + 2A - 2E" - 2N)(E U + TV), 
-s(E r + 2E* - /j,)dr = -N(3N - 3 + 2E V - /i)u, 

\rsd 2 r = i/V(JV - l)u, 

r(n + 2fi-2- 2E t )d t = (n + 2u- 2E" - 2N)(d v + —), 

v 

-r(E r + 2E S - \)d r = -N(2E U - A + 3N - 3) (26) 
when acting on r N p(u, v). Taken all together, we get 
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Lemma 4.3 The substitution i25V transforms the former system ofPDE's 
to a hypergeometric differential operator 

P^ V (X,H) = ^N(N -l)uv + {n + 2X-2E u - 2N)(E U + N) 

-iV(37V - 3 + 2E V - fiju + —N(N - l)u 

N 

+(n + 2/i - 2E" - 2N)(d v + — ) - N(2E U — A + 3N — 3), 

i> 

(27) 

fulfilling 

In the next subsection we find, for generic values of the inducing pa- 
rameters A, fi, a unique solution for this hypergeometric equation for a 
given homogeneity. 

4.3 Solution of the hypergeometric differential equa- 
tion in two variables 

We start with a couple of simple examples. 

Example 4.4 Let us consider the polynomial of homogeneity one, 

p(r, s, t) = Ar + Bs + Ct, A,B,C G C. 
The application of P[' 3 ' (A, fj.) yields 

ff M (A, n)(Ar + Bs + Ct) = &(B(n + 2A - 2) + Afi) + Vi {C{n + 2^-2)+ AX) (28) 

for all i = 1, , n. When A is normalized to be equal to 1, we get 

c = A B = (± 

n + 2/^-2' n + 2A-2' 

The unique homogeneous (resp. non-homogeneous) solution of P[' s ' t (X, /z) 
for all i = 1, . . . ,n is then 

p(r,s,t) = (n + 2A-2)(n + 2/i-2)r-/i(n + 2|Lt-2)s- A(n + 2A-2)t. (29) 

Example 4.5 Let 

p(r, s, t) = Ar 2 + Bs 2 + Ct 2 + Drs + Est + Frt, A, B, C, D, E, F e C 

be a general polynomial of homogeneity two. We have 

P[- S -\X, fi)p(r, s, t) = £i\r(D{n + 2A - 2) + A2(p - 1)) + s(B2(n + 2A - 4) + Dfi) 
+t(A + E(n + 2A - 2) + F(p, - 2))], 

+Vi[r(F(n + 2fi - 2) + A2(A - 1)) + a (A + E(n + 2^-2) + D(X - 2)) 

+i(C2(n + 2^ - 4) + FA)] , (30) 

for all i = 1, . . . , n. The equations 

^P^A.m) = 0, ^UiP^iX,!!) = 
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are equivalent to two systems of linear equations: 

D(n + 2A - 2) + A2((i - 1) + A + E(n + 2(i - 2) + D(\ - 2) = 0, 

F(n + 2(i-2) + A2(X - 1) = 0, 

B2(n + 2(i - 4) + D(i = 0, 

A + E(n + 2\ - 2) + F(n - 2) = 0, 

C2(n + 2fi - 4) + FA) = 0, (31) 

resp. 

D(n + 2A - 2) + A2(^ - 1) = 0, 
B2(n + 2(i - 4) + D(i = 0, 

A + E(n + 2A - 2) + F(/i - 2) + F(n + 2(i-2) + A2{\ - 1) = 0, 
A + E(n + 2(i-2) + D(X - 2) = 0, 

C2(n + 2(i-4)+F\ = 0. (32) 
Both systems are equivalent under the involution 

A^A,E< — > E, D i — > F, B < — > C, X ^ (i 
and its unique solution invariant under this involution is 

A=1,F= - 2{X ~ 1 lc= A(A - 1} 



n + 2^-2' (n + 2^-2)(n + 2^-4) : 
g _ 2 (A-2)(/i-2)-(l + §) d _ -2( M -1) 



(n + 2/x-2)(n + 2A-2)' (n + 2A - 2) ' 

m(m - 1) 



(n + 2A-2)(n + 2A-4)' 



(33) 



T/ie vector 



p(r,s,t) = (n + 2A-2)(ra + 2A-4)(n + 2/ i -2)(n + 2 / it-4)r 2 
+//(// - l)(n + 2(i - 2)(n + 2/i - 4)s 2 
+A(A - l)(n + 2A - 2)(n + 2A - 4)i 2 
-2(/x - l)(n + 2A - 4)(n + 2(i - 2)(n + 2/i - 4)rs 
+2((A - 2)(/x - 2) - (1 + -))(n + 2A - 4)(n + 2(i- 4)st 
-2(A - l)(n + 2A - 2)(n + 2A - 4)(n + 2(i - 4)rt. (34) 

is then the unique solution of P[' 3 ' (A, (i) of homogeneity two. 

We now return back to the situation of a general homogeneity. The 
dehomogenesation (r, s, t, ) — > (r, w, w) is governed by coordinate change 

u := -, v := -, r := r, (35) 
r r 



and so 



d s -¥ -d u , d t -¥ -d v , d r -¥ ~-ud u ~ -vd v + d r , 
r r r r 

E' ► E", E ! ► E", E' ► E" E" E r . (36) 
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The terms in P^' s,t (A,/i) transform into 

^stdr -> ^uv{E u +E V -E r + 1)(E"+E 1, -E r ), 
-r(E r + 2E S - X)d r > (E" + E° - E r )(E" — E" + E r — A — 1), 
r(n + 2/i - 2 - 2E*)3< -> (n + 2^ - 2 - 2E I, )<9 I) , 

irsd 2 iu(E" + E" - E r + 1)(E U + E v - E r ), 

-s(E r + 2E* - v)d r > u(E" + E° - E r )(-E" + E" + E r - ^ - 1), 
s(n + 2A -2 -2E s )d s -)■ (n + 2A - 2E")E U , (37) 

and when acting on a polynomial of homogeneity TV, p(r, s, t) — r N p(u, v) 
for a polynomial p(u, v) of degree TV in u, v, E' = TV and we get 

ff"(A,At) = iut;(E u + E" — TV + 1) (E 11 + E" — TV) 

— (E 11 ) 2 + E u {n + A - 1) + (E v - N)(-E v + TV - A - 1) 
+ (n + 2^-2-2E")d„ 

+ i«(E" H-E* - TV)(-E" + 3E" + TV - 2/x - 1). (38) 
Similarly, one gets 

P"^(A,/Lt) = iu«(E"+E"-TV + l)(E 11 + E"-TV) 

— (E") 2 + E"(n + /* - 1) + {E u - N)(-E u +N-/J.-1) 
+ (n + 2A - 2 - 2E")<9 U 

+ i«(E 1 ' +E" - JV)(-E" + 3E" + TV - 2A - 1). (39) 

Let us denote Aij(X,/j.) the coefficient by monomial tfv 3 in the poly- 
nomial p(u,v). The assumption Aij{X, fi) = Aj t i(fj,, A), combined with 
the symmetry between P^ ,v (X,fi) and P™' v (X, fi), allows to restrict to the 
action of P^ 1,t '(A,/i) on a polynomial of degree TV of the form 

p(u,v)= ^i,j(A,M)wV, =Aj,i(n,\), (40) 

i,j\i<N, j<N 

thereby converting the differential equation (|38|) into the four-term func- 
tional relation 

l(i + j-N-l)(i+j-N- 2)Ai_i,j_i(A, /i) 

+{-i 2 +i[n + A - 1) + (j — N)(—j + TV — A — l)Ai, 3 (A, /x) 
+(j + l)(n + 2/x - 2 - 2j)A !J+1 (A, M ) 

+ i(i+j-TV-l)(-i + 3i + TV-2/z)A i _i,j(A, M ) = (41) 

for i, j = 1, . . . , TV and j > i, which recursively computes Ai,j +1 (X, fi) in 
terms of Ai-ij-i (A, /i), Ai-\,j (A, /i) and Ai, 3 *(A, /it). 

There is still one question we have not mentioned yet, concerning 
the normalization of Aij (A, //). A singular vector can be normalized by 
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multiplication by common denominator resulting in the coefficients valued 
in Pol[X, fj] rather than the field C(A, fJ,). As we shall prove in the next 
Theorem, a consequence of (|41|l is the uniqueness of its solution in the 
range A, /i ^ {m — ^\m £ N}. We observe that the uniqueness of solution 
fails for A, fj, G {m - ||m £ N}, which indicates the appearance of a non- 
trivial composition structure in the branching problem for generalized 
Verma modules. 

In the following Theorem we construct a set of singular vectors, which 
will be the representatives realizing abstract character formulas of the 
diagonal branching problem in Corollary 13.21 

Theorem 4.6 Let A, fi G C \ {m - § \m £ N}, N G N, and (x)i = 
x(x + l) . . . (x + l — 1), / G N be the Pochhamer symbol for x G C. The four- 
term functional equation f or the set {-Ai,j(A,/i)}j l j e {i j ... j jv} fulfilling 

Aj,i(\,n) = Aij(n,\), j > i, 

has a unique nontrivial solution 

^j(A,/i) = 

r{i + j - Apr(i - | - M )r(i - 1 + j - n + A)r(A + %-i) 
2 i+j (-y+H\j\ r(-7V)r(i - n + A)r(i + j - § - p)r(A + § ) ' 

S (-)* (l) W - i + 1 + fc )- fc ( A + ? - Oi-k(/* - iV + x ) fc ( A - N + 1 



Om£ o/ t/ie range A,(igC \ {m - |jm 6 N}, J^gp is sti// a solution, but 

not necessarily unique. 

Proof: 

Let us first discuss the uniqueness of the solution. The knowledge of 
Ai i j(X,^i) for i + j < ko allows to compute the coefficient Aij+i(X, fi) 
with i + j — ko + 1 from the recursive functional equation, because of 
assumption A, /i ^ {rn — f |m G N}. The symmetry condition for Aij (A, fi) 
gives Aj t i(/j,, A) = Ai : j(X, and the induction proceeds by passing to the 
computation of Ai : j+z(X, fi). Note that all coefficients are proportional to 
j4o,o(A,/i) and its choice affects their explicit form. 

The proof of the explicit form for Aij(X, /i) is based on the verification 
of the recursion functional equation (|41|l . To prove that the left hand side 
of (|41|l is trivial is equivalent to the following check: up to a product of 
linear factors coming from T-functions, the left hand side is the sum of four 
polynomials in A, fi. A simple criterion for the triviality of a polynomial 
of degree d we use is that it has d roots (counted with multiplicity) and 
the leading monomial in a corresponding variable has coefficient zero. 

It is straightforward but tedious to check that the left hand side of 
(141 p has, as a polynomial in A, the roots A = k — ^ for k = 1, . . . , i and 
its leading coefficient is zero. Let us first consider A = i — ^ , so get after 
substitution 

AiAi _n tfl) _(-y(i + j-N-l)...(-N) 



2 ,rv 2*+3'i!j! 

. g _ /VWu - N + 11, 

(43) 



(j - N - | ) . . . (i - N - | + 1)(1 - f - jVQiQx - N + 1) 



- M )...(l-f -/z)(A+§ (A _i) 
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and 



Taken together, there remain just two contributions on the left hand side 
of (|41|) given by Aij(i— ?,/*), §,/tt). Up to a common rational 

factor, their sum is proportional to 

a+1)( .^-,- w -, <^y^a -, 

which proves the claim. The proof of triviality of the left hand side at 
special values A = £ — 1 — — §is completely analogous. 

Note that there are some other equally convenient choices for A, /i 
allowing the triviality check for (|4ip . for example based on the choice 
A = k + N — 1, k — 1, . . . ,i or fj, = N — k, k — 1, . . . ,i. 

The remaining task is to find the leading coefficient on the left hand 
side of (|4ip as a polynomial in A. Because 

(A+g-Oi-jk^A 4 -*, 

(X-N+l-k) k A ~°° A fc , (45) 

the polynomial is of degree A- 7-1 ~ = A J_l , j > i. The leading coefficient 
of Aij(X, fi) is 



A 



(~) i+J "(» + J - JV - 1) . . . (-JV) 
2 ! +^!j! (i - f - /i) • • • (1 - f - A*) ' 

There are three contributions to ()41|) : 

Ai,j (A, /x) 



(46) 



(N —j + i) lim 



A-+cx) A- 7 ' - * 
(j + l)(n + 2 M - 2 - 2j) lim 

A— >-oo A- 7 ^ 

i(i + j - N - l)(-i + 3j + iV - 2/*) lim ^-^y , (47) 

whose sum is a polynomial in /i multiplied by common product of linear 
polynomial. In order to prove triviality of this polynomial, it suffices as 
in the first part of the proof to find sufficient amount of its roots and 
to prove the triviality of its leading coefficient. For example in the case 
/x = N — 1, we get from (|47|) that the coefficients of this polynomial are 
proportional to the sum 

( AT ■• , | (J + m + 3-N) + Sj + N- 2(N - 1)) 

( J+)+ U-i + i) U-i + i) 
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which equals to zero. The verification of the required property for /i = 
JV — k, k = 2, . . . , i is completely analogous. This completes the proof. 

□ 

This completes the description of the set SoI(q(Bq, diag(g), Ca, m ) char- 
acterizing solution space of a diagonal branching problem for so(n + 
1,1, R), @. 

Remark 4.7 It is an interesting observation that the four term func- 
tional equation \4^\l f or Aij(X,fi) can be simplified using hypergeometric 
functions 3F2: 



3^2(01, 02,03; bi, 62; z) := y j 



(ai) m (02)m(a3) 

(&l)m(b 2 )m ml' 



where 01, «2, 03, bi, 62 £ C and (») m = cc(a; + 1) . . . (x + m — 1). Jn partic- 
ular, it can be converted into four term functional equation 

(n + 2X)T(i +j- jV)T(-| - A)r(l - i + j - N + A)r(l - f - /x) 

2 i +i(-) i +JT(i + i)r(-iv)r(i - jv + A)r(i + j - f - n) 



Tl 

{i{~2j + n + 2^)3^2(1 - i, AT - A, 1 - JV + n; 1 - i + j, 1 - - - A; 1) + 

+ i - j + TV - A)(i - 3? - JV + 2/*) ■ 
• 3 .Fa(l - »> N - A, 1 - N + fi; 2 - i + j, 1 - | - A; 1) + 
{i 2 - + n + A) + (j - JV)(1 + j - JV + A)) • 
•3F2H, JV - A, 1 - JV + /x; 1 - i + j, 1 - ~ - A; 1) + 
(1 + + 3 ~ JV)(-1 + i - j + JV - A) • 

• 3 F 2 (-i, JV - A, 1 - JV + w 2 - i + j, 1 - '- - A; 1)) = 0. (48) 

T/iis functional equation, whose knowledge would clearly simplify the for- 
mulation of the proof of the last Theorem, is advanced to be found in any 
standard textbook on special function theory of several variables (see e.g., 
HF, £ZFA an d does not seem to be accessible in the literature. 

In fact, a large part of the monograph fll is devoted to evaluations of 
generalized hypergeometric functions P +iF p at z — 1, at least for reason- 
ably small values of p G N. However, the Saalschutz's theorems of the 
form 

y> (ggMg + |g - b)r(-4) r (a + 2r) n _ r 
^ r\(n - r)\(l + a - b) r 

^3*2(~ + -a - 6, a + n, -n; 1 + a - b, ~ + Ja; 1) (49) 
n! 2 2 2 2 

are too special and restrictive to be of direct use for our needs. 
Example 4.8 As an example, we have 



r(i + , ? -JV)r(j-iv + A)r(i-f - M ) 
2j+i(_)j+ir(i + j)r(-jv)r(i - jv + A)r(i + j - § - ^ 

(j(-2 + n + 2\) + 2{N - A)(l — JV + 
(n + 2A - 2) 



(50) 



17 



for ail jS {!,..., N}. 



Let us also remark that for special values A,/i G {m — §|m € N}, the 
formula Aij(X,ii) simplifies due to the factorization of the underlying 
polynomial. Our experience suggests that the factorization indicates so 
called factorization identity, when a homomorphism of generalized Verma 
modules quotients through a homomorphism of generalized Verma mod- 
ules of one of its summands (in the source) or a target homomorphism of 
generalized Verma modules. This naturally leads to the question of full 
composition structure of the branching problem going beyond the formu- 
lation in terms of the Grothendieck group of Bernstein-Gelfand-Gelfand 
parabolic category O. 

Let us mention another interesting observation. The diagonal coeffi- 
cients Ai t i(\, fj.) = Ai t i(fj,, A) are, up to a rational multiple coming from the 
ratio of the product of F-functions, symmetric with respect to A o fi. As 
a consequence, these polynomials belong to the algebra of ^-invariants: 

C[A, M ] Z2 ^C[Xn,X + lA- 



Lemma 4.9 The diagonal coefficients can be written as 

r(2i - N) 



2^+H\ r(-JV)r(l n)T{\ + i - f - A) 



71 71 71 

(r(l + i - - - A)r(l - - - /i)sFa(-t, N - A, 1 - N + p, 1, 1 - - - A; 1) + 

71 71 71 

r(l + i - - - M )r(l - - - X) 3 F 2 (-i, 1 - N + A, N - n; 1, 1 - - - M ; 1)) 
Proof: 

It follows from the definition of 3F2 that 
1 

r(i-H- I -fi)r(i + i- I - A) ' 

71 71 71 

(r(l + i - - - A)r(l - - - n) 3 F a (-i, N - A, 1 - N + p, 1, 1 - - - A; 1) + 

71 71 77 

r(l + t - - - mMI - - - A) 3 F 2 (-i, 1 - N + A, AT - w 1, 1 - - - M ; 1)) = 
(-i) m {N - \) m (l - N + n) m 



_ '(lm)(l-t-A) m (l-§-Ai)i-l 

H) ro (JV-/x) m (l-JV + A) m . 1 . . 

(lm)(l-?-A*)m(l-§-A)i_i ; m!' 1 j 

where the sum is finite due to the presence of (— i) m in the nominator. 
Using basic properties of the Pochhamer symbol, e.g. (x) m = ( — ) m (— x + 
m — l) m , an elementary manipulation yields the result. 

□ 

Example 4.10 As an example, in the case of i — 1 we have 

N(N - 1)(A M - 7V(A + g) + (1 - § + N(N - 1))) 
Am ( A ' M) = (2A + n-2)(2 M + n-2) (52) 
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Let us summarize our results in 

Theorem 4.11 Let g = so(n + 1, 1,R) be a simple Lie algebra and p 
its conformal parabolic subalgebra with commutative nilradical. Then the 
diagonal branching problem for the scalar generalized Verma U(g © 
modules induced from character (A, fi) is determined in the Grothendieck 
group of Bernstein- Gelf and- Gelf and parabolic category O p by bt(g) -isomorphism 

oo 

A+ M -2j(fl,p)- (53) 

2=0 

Here the summand Mx+fi-2j(s, p) is generated by singular vector of ho- 
mogeneity 2j and the form H25\) with coefficients given by eguation \4>ty > 
j £ N. In particular, the singular vectors are non-zero, unique up to a 
multiple, linearly independent and of expected weight ( given by homogene- 
ity), and the cardinality of the set of singular vectors is as predicted by 
Corollary\3J% 

The explicit formulas for the singular vectors will be given, in the dual 
language of bilinear differential operators, in the next section. 

5 Application - the classification of bilin- 
ear conformal invariant differential opera- 
tors on line bundles 

Let M be a smooth (or complex) manifold equipped with the filtration of 
its tangent bundle 

C T 1 M C • • • C T m °M = TM, 

V — > M a smooth (or holomorphic) vector bundle on M and J k V — > M 
the weighted jet bundle over M, defined by 

J k V= U J*V, J k x V ^ ®tiHom{U l {gr{T x M)),V{V x )), 

where Ui(gr(T x M)) is the subspace of homogeneity fc-elements in the uni- 
versal enveloping algebra of the associated graded gr(T x M). A bilinear 
differential pairing between sections of the bundle V and sections of the 
bundle W to sections of the bundle y is a vector bundle homomorphism 

B : J k V (g> J l W — > y. (54) 

In the case M — G/P is a generalized flag manifold, a pairing is called 
invariant if it commutes with the action of G on local sections of the 
homogeneous vector bundles V, W, 3^- Denoting V, W, Y the inducing 
P-representations of homogeneous vector bundles V, W, y, the space of 
G-invariant differential pairings can be algebraically characterized as 

((W(g) ® W(g)) ® W (p)® M( p) Hom(Y ® W, Y)f ~ 
Hom uQ)) (M(s,p,Y),M(s®~s,p ©p,(V v ®W V )), (55) 

where the superscript denotes the space of P- invariant elements and V v , W v 
denote the dual representations, see e.g., [IS]. In our case, we get 
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Theorem 5.1 Let G = S0 o (n + 1, 1,R) and P its conformal parabolic 
subgroup, A, /i G C\ {m — j\m € N}, TV € N. Let us denote by C-\ 
the homogeneous line bundle on n- dimensional sphere G/P ~ S n induced 
from the character X—X of P. Then there exists up to a multiple a unique 
set of bilinear conformally invariant operators 

B N : C*°°(G/P,£_ A ) x C 00 (G/P,£- ll )->G ao (G/P,C-x- ll -air) (56) 
of the form 

B N = Y A i j{X i (t)S l i i f k , (57) 



0<i,j,fc<JV|i+j + fc = iV 



where 



r(i + j- /v)r(i - § - At )r(i -i + j-N + \)r{\ + 
2 i +i(-y+H\j\T(-N)T{l-N + X)r{l+j - | -n)r{\+ 1) ' 

( * J (i - < + 1 + + | - Oi-kO* - + l) fc (A - TV + 1 

^<,j(A,Aj) = Aj ti (fi, A), 

n n n 

• 5 = E a i = A -' * = E d y> = A y>? = I2 d *^- ( 59 ) 

i=l i — 1 i—1 

Out of the range A, fj, £ C\ {m - ||m 6 N}, Sat is stiii an element of the 
previous set, but there might be additional ones indicating the emergence 
of a nontrivial composition structure. 

Proof: 

The proof is a direct consequence of Theorem [42] and duality (|55[) . 
together with the application of inverse Fourier transform 

xj i — ► -id (j , d Xj i — ► 

with i £ C the imaginary complex unit. 

□ 

In many applications, it is perhaps more convenient to express the 
bilinear differential operators in terms of tangent resp. normal coordi- 
nates U = + Vi) resp. ni — §(£i — Vi), i = 1, . . . ,n to the diagonal 
submanifold, where 

r = -(< t,t > — < n, n >), 

s = -(< t,t > + < n,n > +2 < t,n >), 

t = -(<t,t> + <n,n> -2<t,n>). (60) 
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6 Open problems and questions 



It would be interesting to find the composition structure appearing for 
integral values of inducing representation-theoretical parameters, a ques- 
tion closely related to the factorization structure of special polynomials 
which are solutions of the problem. Another task is the extension of the 
present results to the case of vector valued representations, and also to the 
other couples (g, p) of a simple Lie algebra and its parabolic subalgebra of 
interest. This sounds clearly as a program rather than several unrelated 
questions. 
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